This is a ‘strong opinion, weakly held’.

Code produced by LLMs becomes unmaintainable except by other LLMs. This is because of the rise of cognitive debt, whereby code is generated at a rate faster than it becomes human-legible. This creates a kind of perverse incentive that encourages using LLMs for further code contributions, because this requires interfacing with code that has been generated by AI, and has yet to become human legible. In the limit, humans end up contributing no code at all. Half human-half AI code contribution is not a stable equilibrium.

This isn’t a claim at the individual level, it is a claim at the group level. There will be some code owners who stringently enforce that a human has read and understood every line of code produced by an AI agent. But this will be a rarity. Why? Because almost all of our incentives are based on the end artefact, not the code that drives it. If you can, with high probability, guarantee an end product that matches user requirements (security, performance, quality, etc.), you do not require human legibility of the codebase that underpins it. To be clear, it might matter to some, but it won’t matter on the scale of society-wide incentives (chief among them being profit).

This claim is dependent on the phrase “with high probability”. In particular, this threshold for “high probability” is greater than the probability afforded by a human doing the equivalent task - we demand more from AI because there is a reduced element of culpability. If you believe, as many do, that AI capabilities will generally progress to the point where this notion of “high probability” is satisfied, then the rest of this claim should feel trivially obvious. There are already indications that this threshold will soon be satisfied for one of the most important vestiges of human involvement at the level of code: security.

While there may be some new paradigm developed. I do not yet know what this is. Perhaps there will be analogies to Test Driven Development (TDD), where humans mostly focus on setting project requirements in natural language, and agents handle everything else. Where exactly humans end up on this totem pole, will likely have less to do with where they feel comfortable, and more to do with where they can still contribute meaningfully to these macro-scale incentives.

–

Post-script

The far more interesting question here, is whether this claim generalises beyond code - to the real world. If AI can produce positive-outcome decisions at a rate faster than those decisions can become legible for humans, it is worthwhile to ask what the stable long-term equilibrium looks like in this case.